An Analysis of Contrastive Divergence Learning in Gaussian Boltzmann Machines
نویسندگان
چکیده
The Boltzmann machine (BM) learning rule for random field models with latent variables can be problematic to use in practice. These problems have (at least partially) been attributed to the negative phase in BM learning where a Gibbs sampling chain should be run to equilibrium. Hinton (1999, 2000) has introduced an alternative called contrastive divergence (CD) learning where the chain is run for only 1 step. In this paper we analyse the mean and variance of the parameter update obtained after i steps of Gibbs sampling for a simple Gaussian BM. For this model our analysis shows that CD learning produces (as expected) a biased estimate of the true parameter update. We also show that the variance does usually increase with i and quantify this behaviour.
منابع مشابه
Average Contrastive Divergence for Training Restricted Boltzmann Machines
This paper studies contrastive divergence (CD) learning algorithm and proposes a new algorithm for training restricted Boltzmann machines (RBMs). We derive that CD is a biased estimator of the log-likelihood gradient method and make an analysis of the bias. Meanwhile, we propose a new learning algorithm called average contrastive divergence (ACD) for training RBMs. It is an improved CD algorith...
متن کاملRestricted Boltzmann Machines with Gaussian Visible Units Guided by Pairwise Constraints
Restricted Boltzmann machines (RBMs) and their variants are usually trained by contrastive divergence (CD) learning, but the training procedure is an unsupervised learning approach, without any guidances of the background knowledge. To enhance the expression ability of traditional RBMs, in this paper, we propose pairwise constraints restricted Boltzmann machine with Gaussian visible units (pcGR...
متن کاملStochastic Gradient Estimate Variance in Contrastive Divergence and Persistent Contrastive Divergence
Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known tha...
متن کاملInductive Principles for Learning Restricted Boltzmann Machines (DRAFT: August 25, 2010)
We explore the training and usage of the Restricted Boltzmann Machine for unsupervised feature extraction. We investigate the many different aspects involved in their training, and by applying the concept of iterate averaging we show that it is possible to greatly improve on state of the art algorithms. We also derive estimators based on the principles of pseudo-likelihood, ratio matching, and ...
متن کاملData Normalization in the Learning of Restricted Boltzmann Machines
In practice, training Restricted Boltzmann Machines with Contrastive Divergence and other approximate maximum likelihood methods works well on data with black backgrounds. However, when using inverted images for training, learning is typically much worse. In this paper, we propose a very simple yet very effective solution to this problem. The new algorithm requires the addition of only three(!)...
متن کامل